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FOR  DIRECT  NUMERICAL  SIMULATIONS  (DNS)  AND  LARGE-EDDY- 
SIMULATIONS  OF  TRANSITIONAL  AND  TURBULENT  FLOWS 


Final  Report 

by 

Hermann  F.  Fasel 


Abstract 

With  the  funding  from  DURIP  grant  F  46920-97-1-0812  we  purchased  a  Silicon  Graphics  Origin 
2000  with  12  processors,  2GB  memory,  and  18GB  disk  space,  as  well  as  flow  visualization  soft¬ 
ware  and  several  peripherals,  accessories  and  PC’s  for  pre-and  post-processing  support.  The 
equipment  has  been  utilized  effectively  to  perform  pre-  and  post-processing  tasks  for  our  DOD 
funded  research  projects.  The  multi-processor  capability  has  allowed  us  to  develop  and  test  our 
Navier-Stokes  codes  locally  and  improve  their  performance  before  performing  production  runs  at 
the  DOD  High-Performance  Computing  Centers.  The  high  performance  multi-processing  capa¬ 
bility  has  enabled  effective  post-processing  of  the  massive  time-dependent  data  sets  that  are  de¬ 
livered  by  the  simulations  on  the  DOD  supercomputers. 


Equipment  purchased  with  funding  from  DURIP  grant  F  49620-97-1-0812: 

Silicon  Graphics  Workstations  $273,776.26 

Compute  Server 

[Origin  2000  by  Silicon  Graphics  Inc.] 

Including:  Origin  2000  Rack 

2  Insert  Modules  w/  3  Node  Boards  each 

12  R10000/195MHz  CPUs  w/4MB  secondary  cache 

2  GB  Memory 

4.5  GB  System  Disk 

2  9GB  Ultra  SCSI  Disks 

Irix  6.4  Operating  System 

Installation 


PTIG 


J352EXSSED 
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SZIQ6661 
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Peripherals,  Accessories,  and  Software  $31,425.88 

Color  Laserjet  Printer 

[HP  Color  Laserjet  5  by  Hewlett  Packard,  Sehi  Computer  Products,  Inc.] 
Including:  HP  Color  LaserJet  5  Printer 

Postscript  SIMM 
32  MB  Memory 
Rear  Feed  Unit 

DLT  Tape  Drive 

[External  DLT  7000  Tape  Drive  by  Nordisk  Systems,  Inc.] 

Accessories 

[Nordisk,  CIC  Systems,  Electronic  City] 

Including:  Data  Cartridges 

Printer  Supplies 
Static  Mat  and  Wrist  Strap 


Software 

[Silicon  Graphics,  Wolfram  Research,  Advanced  Visual  Systems] 
Including:  Silicon  Graphics  Varsity  Software  Licenses 

Mathematica 

A  VS  Educational  Developers  License 


Personal  Computers  $6,858.64 

Desktop  PC 

[Dell  Dimension  XPS  by  Dell  Computer] 

Including:  Dell  Dimension  XPS  Desktop  Computer 

Intel  Pentium  II  400MHz  CPU 
64MB  RAM 
8GB  System  Disk 
32x  CD  ROM  Drive 
IOMEGA  ZIP  Drive 
33.6/56K  Winmodem 

Desktop  PC 

[GP5-166  Desktop  by  Gateway  2000] 

Including:  GP5-166  Desktop  Computer 

Intel  Pentium  166  MHz  CPU 
64  MB  RAM 
4  GB  System  Disk 
CD-ROM  Drive 
10/100  Ethernet  Card 
17"  Monitor 
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Notebook  PC 

[Solo  2300  SE  by  Gateway  2000] 

Including:  Solo  2300  SE  Notebook  Computer 

Intel  Pentium  133  MHz  CPU 
32  MB  RAM 
2  GB  System  Disk 
Modular  CD-ROM  Drive 
33.6  PCMCIA  Fax/Modem 
Ethernet  Card 


TOTAL  COST 


$312,060.78 


University  of  Arizona  Cost  Sharing  $67,784.00 

DOD  COST  $244,276.78 


Summaries  of  Research  Projects  for  which  Equipment  Was  Used 

DOD  Agency:  Air  Force  Office  of  Scientific  Research 
Contract  Number:  F49620-97- 1-0208  (Dr.  Mark  Glauser) 

Title:  Control  of  Separation  Using  Pulsed  Wall  Jets:  Numerical  Investigations  Using  DNS  and  LES 
Principal  Investigator:  H.  Fasel 
Duration:  May  1,  1997-December  31,  1999 
Amount:  $214,279 

Wall  jets,  or  wall  jet-like,  flows  are  of  great  technical  relevance  for  numerous  aerospace  applica¬ 
tions.  Wall  jets  can  be  used  as  efficient  means  of  controlling  external  and  internal  boundary  layers.  For 
external  applications,  they  are  used,  for  example,  for  boundary  layer  control  on  airfoils.  Experiments  by 
Wygnanski  et  al.  (1996)  have  demonstrated  the  effectiveness  of  pulsed  wall  jets  to  control  separation  for 
flows  over  single-element  or  segmented  airfoils.  For  gaining  insight  into  the  fundamental  mechanisms 
responsible  for  the  often  striking  effect  of  periodic  forcing  on  wall  jets,  forced  transitional  and  turbulent 
wall  jets  are  being  investigated  in  the  present  research  project  using  Direct  Numerical  Simulation  (DNS) 
and  Large-Eddy  Simulation  (LES).  While  use  of  DNS  and  LES  for  flows  over  actual  slotted  flaps  are  the 
final  goal  of  our  research  effort,  several  intermediate  steps  are  taken  to  explore  different  aspects  of  this 
complex  flow  geometry.  Clearly,  for  wall  jets  over  actual  segmented  airfoils,  the  combined  effects  of  ad¬ 
verse  pressure  gradient  and  curvature  plays  a  major  role.  To  isolate  the  relevant  mechanisms,  this  com¬ 
plex  flow  geometry  is  first  broken  down  into  simpler  modules  so  that  the  effect  of  pressure  gradient  and 
curvature  can  be  investigated  separately.  Both  DNS  and  LES  are  performed  using  a  three-dimensional 
Navier-Stokes  code  based  on  an  incompressible  vorticity-velocity  formulation.  For  the  time  integration,  a 
fourth-order  Runge-Kutta  method  is  employed.  For  the  spatial  discretization  in  the  streamwise  and  the 
wall-normal  directions,  fourth-order  accurate  compact  differences  are  used,  while  the  spanwise  direction 
is  treated  pseudo-spectrally.  For  LES  of  turbulent  wall  jets,  a  Smagorinsky-type  subgrid-scale  turbulence 
model  has  been  incorporated  into  the  code.  More  sophisticated  subgrid-scale  models  are  currently  being 
implemented.  In  our  LES  of  transitional  and  turbulent  wall  jets,  the  turbulent  flow  is  periodically  forced 
using  a  blowing  and  suction  slot  in  the  wall.  The  effect  of  the  forcing  is  investigated  by  analyzing  the 
time-dependent  flow  data  using  statistical  methods,  Fourier  decomposition  in  time,  and  flow  visualization 
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including  tracking  of  the  large  coherent  structures  that  are  generated  due  to  the  forcing.  While  the  actual 
simulations  are  being  performed  remotely  on  DOD  supercomputers,  analysis  and  visualization  of  the 
massive  amount  of  unsteady  flow  data  can  now  be  performed  locally  with  the  help  of  the  acquired  equip¬ 
ment. 


DOD  Agency:  Army  Research  Office 

Contract  Number:  DAAG55-97-1-0128  (Dr.  Thomas  Doligalski) 

Title:  Passive  and  Active  Control  of  Supersonic  Axisymmetric  Base  Flows:  Numerical  Investiga¬ 
tions  Using  Direct  Numerical  Simulations  and  Large-Eddy  Simulations 
Principal  Investigator:  H.  Fasel 
Duration:  May  I,  1997-April  30,  2000 
Amount:  $217,612 

A  comprehensive  effort  is  being  undertaken  to  investigate  transitional  and  turbulent  axisymmetric 
wakes  behind  cylindrical  bodies  aligned  with  the  flow  at  supersonic  speeds.  Particular  emphasis  is  on 
identifying  and  understanding  the  dynamical  behavior  of  the  large-scale  vortical  structures  that  control  the 
flow  behavior  in  a  supersonic  wake.  Direct  Numerical  Simulations  (DNS)  and  Large-Eddy  Simulations 
(LES)  are  the  main  investigative  tools.  The  numerical  simulations  are  supported  and  complemented  by  a 
theoretical  effort  based  on  stability  theory  analysis  and  Proper  Orthogonal  Decomposition  (POD)  tech¬ 
niques,  which  are  applied  to  the  numerically  generated  data. 

For  axisymmetric  aerodynamic  bodies  in  supersonic  flight,  the  flow  field  in  the  wake  region  has  consider¬ 
able  effect  on  the  aerodynamic  drag.  Even  small  changes  in  the  flow  behavior  of  the  wake  may  affect  the 
performance  of  the  entire  flight  vehicle,  e.g.,  missiles,  rockets,  or  projectiles.  The  effect  on  the  aerody¬ 
namic  drag  is  mainly  due  to  the  recirculation  region  that  develops  in  the  base  region  of  the  body  and  thus 
to  the  low  pressure  associated  with  the  recirculating  flow  (base  drag).  Flight  tests  with  projectiles  (U.S. 
Army  549  projectile)  have  shown  that  the  base  drag  may  account  for  up  to  35%  of  the  total  drag.  This 
suggests  that  attempts  to  modify  the  near-wake  flow  such  that  the  base  pressure  would  increase  could  be 
highly  rewarding  with  respect  to  drag  reduction  and,  as  a  consequence,  with  regard  to  increasing  the  per¬ 
formance  characteristics  of  flight  vehicles  or  projectiles. 

It  is  well  known  that  for  subsonic  (incompressible)  wakes,  the  dynamics  of  the  large  (coherent)  structures 
play  a  dominant  role  in  the  local  and  global  behavior  of  the  flow.  This  evidence  was  found  from  both  ex¬ 
perimental  investigations  and  numerical  simulations  (including  ours)  and  was  confirmed  by  theoretical 
studies.  For  supersonic  speeds,  on  the  other  hand,  very  little  is  yet  known  about  the  dynamical  behavior  of 
turbulent  flows.  This  is  true  for  supersonic  flows  in  general  and  for  axisymmetric  wakes  in  particular. 

Thus,  the  question  arises:  Do  large  structures  play  a  similarly  important  role  for  supersonic  separated 
flows  and  in  particular  for  supersonic  axisymmetric  wakes?  The  answer  to  this  question  is  of  fundamental 
relevance  for  applying  flow  control.  There  are  few  experimental  investigations  that  have  focused  on  this 
issue.  However,  when  looking  at  flow  visualization  pictures  of  supersonic  wake  flows,  distinct  patterns 
with  large-scale  structures  can  be  observed.  For  supersonic  axisymmetric  wakes,  the  mean  flow  structure 
of  the  near-wake  region  is  characterized  by  the  axisymmetric  shear  layer  originating  at  the  sharp  comers 
of  the  blunt  base. 

Supersonic  axisymmetric  wakes  are  extremely  difficult  to  investigate  experimentally.  Wind  tunnel  inter¬ 
ference  and  interference  from  model  support  strongly  affect  the  mean  flow  behavior,  which  may  be  an 
indication  that  this  behavior  might  be  caused  by  the  presence  of  large  coherent  structures.  Because  of  the 
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experimental  difficulties,  numerical  simulations  represent  a  new  alternative  for  investigating  the  compli¬ 
cated  unsteady  flow  phenomena  in  the  supersonic  wake. 

Direct  numerical  simulations  using  the  complete  Navier-Stokes  equations  are  restricted  to  somewhat  low- 
to-moderate  Reynolds  numbers  because  of  the  rapidly  increasing  demands  on  computing  power  as  the 
Reynolds  number  increases.  Therefore  in  the  course  of  this  research,  we  are  working  on  ways  to  drasti¬ 
cally  increase  the  algorithmic  efficiency  of  our  Navier-Stokes  codes.  Toward  this  end,  we  are  also  incor¬ 
porating  the  use  of  massively  parallel  computers.  With  our  own  smaller-scale  parallel  computing  capac¬ 
ity,  we  have  found  that  our  basic  codes  require  relatively  little  overhead  costs  when  run  on  parallel  ma¬ 
chines. 

Because  of  the  Reynolds  number  limitations  on  direct  simulations,  we  are  also  performing  Large-Eddy 
Simulations  (LES)  using  subgrid-scale  turbulence  models.  Since  our  Navier-Stokes  equations  are  highly 
suited  for  implementation  of  subgrid-scale  turbulence  models,  we  are  able  to  use  essentially  identical 
codes  for  both  the  direct  and  large-eddy  simulations.  This  allows  proper  fine-tuning  of  the  LES  codes  (in 
particular,  of  the  subgrid-scale  models)  so  that  they  can  be  applied  with  greater  confidence  to  the  higher 
Reynolds  number  calculations.  Even  with  current  LES  models,  reproducing  the  large  Reynolds  numbers 
seen  in  experiments  requires  a  massive  computational  effort.  For  this  reason,  a  new  LES  methodology  is 
currently  being  implemented.  It  will  allow  the  simulations  to  be  pushed  to  considerably  larger  Reynolds 
numbers  than  traditional  LES,  and  thus  closer  to  the  flow  conditions  seen  in  experiments,  while  at  the 
same  time  capturing  the  large  (coherent)  structures  that  in  all  likelihood  have  a  central  importance  to  the 
flow  and  are  not  captured  with  a  Reynolds-averaged  simulation. 


DOD  Agency:  Army  Research  Office 

Contract  Number:  DAAH04-96-1-01 12  (Dr.  Thomas  Doligalski) 

Title:  Numerical  Investigations  of  the  Control  of  Separation  by  Periodic  Oscillations:  Effects  of 
Compressibility 

Principal  Investigator:  H.  Fasel 

Duration:  May  1,  1996-April  30,  1999 

Amount:  $130,000 

Boundary  layer  separation  is  a  critical  limiting  design  factor  for  a  wide  range  of  flow  devices.  For 
example,  separation  drastically  reduces  lift  and  increases  drag  of  airfoils.  It  reduces  the  efficiency  of  noz¬ 
zles  and  turbomachinery.  And,  in  heat  transfer,  separation  can  lead  to  a  local  hot  spot.  Any  means  of  de¬ 
laying  the  onset  of  separation  would  increase  the  design  envelope  for  a  wide  variety  of  engineering  appli¬ 
cations. 

Recent  work  by  Wygnanski  et  al.  (a  survey  of  which  can  be  found  in  Wygnanski  1997)  has  shown  that 
introducing  disturbances  by  periodic  blowing  and  suction  can  delay  or  even  eliminate  separation.  This 
active  control  method  is  much  more  efficient  than  steady  blowing,  since  it  creates  a  net  momentum  flux 
with  no  net  mass  flux.  Also,  since  it  exploits  the  natural  amplification  of  the  control  disturbances  by  the 
flow,  it  requires  a  much  smaller  net  momentum  transfer  to  achieve  the  same  effect.  For  this  reason,  it  is 
applicable  to  a  wide  variety  of  cases  where  steady  blowing  would  be  either  ineffective  or  require  too  high 
a  mass  transfer  rate. 

This  research  attempts  to  extend  these  results  to  high-subsonic  transonic,  and  supersonic  Mach  numbers. 
For  this  research,  we  are  employing  Large-Eddy  Simulations  (LES)  using  subgrid-scale  turbulence  mod¬ 
els  that  are  validated  by  Direct  Numerical  Simulations  (DNS).  For  non-equilibrium  turbulent  flows  like 
the  present  one,  traditional  eddy-viscosity  type  turbulence  models  (such  as  Smagorinsky-based  models) 
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are  insufficient.  Therefore,  in  collaboration  with  C.  Speziale  (Boston  University),  we  are  developing 
more  advanced  two-equation  non-equilibrium  models.  These  models  will  initially  be  used  for  separation 
control  studies  in  flat-plate  adverse  pressure  gradient  boundary  layers,  and  later  in  curved  wall  geome¬ 
tries.  The  data  from  the  numerical  simulations  will  be  analyzed,  focusing  on  understanding  the  dynamical 
behavior  of  the  large  turbulent  structures  when  active  flow  control  by  periodic  forcing  is  employed.  The 
goal  is  to  isolate  the  essential  physical  mechanisms  that  are  responsible  for  the  delay  of  separation. 


DOD  Agency:  Office  of  Naval  Research 

Contract  Number:  N000 14-94- 1-0095  (Dr.  Patrick  Purtell) 

Title:  Effects  of  Adverse  Pressure  Gradient  and  Wall  Curvature  on  the  Turbulence  Mechanisms  in 
Boundary  Layer  Flows:  Reynolds-Averaged  Navier-Stokes  Calculations,  Large-Eddy  Simu¬ 
lations,  and  Direct  Numerical  Simulations  for  the  Stratford  Ramp  /  A  New  Methodology  for 
Numerical  Simulations  of  Complex  Turbulent  Flows:  From  DNS  to  RANS,  a  Combined  Ap¬ 
proach 

Principal  Investigator:  H.  Fasel 

Duration:  October  1,  1993-September  30,  1998 

Amount:  $668,553 

The  main  goal  of  this  research  project  is  to  identify  and  understand  the  combined  effects  of  strong 
adverse  pressure  gradient  and  strong  streamwise  curvature  on  the  turbulence  mechanisms  of  turbulent 
boundary  layers  at  high  Reynolds  numbers.  For  these  investigations,  the  so-called  Stratford  ramp  was 
chosen  as  a  generic  flow  geometry.  For  this  geometry,  both  adverse  pressure  gradient  and  streamwise 
curvature  act  on  the  turbulent  boundary  layer  in  a  unique  way:  the  boundary  layer  is  continuously  on  the 
verge  of  separation.  This  flow  geometry  is  highly  suited  for  our  attempts  to  identify  and  understand  the 
relevant  turbulence  mechanisms  and,  as  a  consequence,  for  developing  a  new  methodology  for  non- 
equilibrium  turbulent  boundary  layers. 

The  new  methodology  is  based  on  employing  a  new  subgrid-scale  (SGS)  turbulence  model  for  time- 
dependent  turbulent  simulations,  where  the  resolved  scales  are  computed  by  solving  the  Navier-Stokes 
equations  and  the  unresolved  scales  are  modeled  by  the  new  turbulence  model.  The  simulations  will  ap¬ 
proach  an  unsteady  Reynolds-Averaged  Navier-Stokes  (RANS)  calculation  when  the  grid  resolution  is 
decreased  (and/or  Reynolds  number  is  increased)  and  consistently  approach  a  Direct  Numerical  Simula¬ 
tion  (DNS)  when  the  grid  resolution  is  increased  (and/or  Reynolds  number  is  decreased).  In  between  these 
limits,  we  have  an  non-traditional  Large-Eddy  Simulation  (LES)  (with  various  degrees  of  modeled  unre¬ 
solved  scales).  The  new  methodology  is  non-traditional  because  the  SGS  model  is  far  superior  to  models 
used  in  traditional  LES,  including  the  so-called  Dynamic  model,  which  have  significant  shortcomings  as 
they  are  based  on  the  Smagorinsky  model  and  are  thus  not  really  applicable  to  complex  geometries.  The 
results  using  the  new  methodology  agree  very  well  with  comparison  calculations.  In  parallel  with  devel¬ 
oping  and  testing  the  new  methodology,  we  carried  out  Direct  Numerical  Simulations  (DNS)  for  the  Strat¬ 
ford  ramp.  In  addition  to  standard  full  blown  DNS  (no  assumptions,  no  modeling),  we  developed  a  sim¬ 
plified  DNS  model  and  performed  extensive  simulations  with  this  model.  From  this  simplified  DNS  we 
found  that  at  a  low  Reynolds  number,  due  to  the  “natural”  generation  of  instability  waves,  the  flow  can 
sustain  a  strong  adverse  pressure  gradient  for  which  the  steady  laminar  flow  would  separate.  However,  at 
higher  Reynolds  number  the  flow  will  separate;  but,  the  separation  can  be  suppressed  by  periodic  forcing. 
The  computational  effort  is  being  conducted  in  direct  and  close  collaboration  with  an  experimental  effort 
by  I.  Wygnanski  (The  University  of  Arizona)  and  a  theoretical  turbulence  modeling  effort  by  C.  Speziale 
(Boston  University).  In  addition,  this  group  effort  is  linked  to  the  computational  effort  by  T.  Huang 
(David  Taylor  Model  Basin),  who  will  test  the  improved  Reynolds  stress  models  that  result  from  our  in¬ 
vestigations  for  Navy  relevant  geometries. 
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Summary  of  How  the  Acquired  Instrumentation  Has  Enhanced  the  DOD-Funded  Research 

The  instrumentation  has  significantly  supported  our  DOD-funded  research  and,  as  a  consequence, 
has  enabled  major  progress  in  DOD-relevant  Computational  Fluid  Dynamics  research  that  would  other¬ 
wise  not  have  been  possible.  In  particular,  with  the  acquired  high-speed  compute  server  (Origin  2000),  we 
were  able  to  close  an  existing  gap  that  had  prevented  us  from  fully  utilizing  the  massive  computing  power 
available  to  us  at  the  DOD  High-Performance  Computing  Centers.  As  summarized  above,  we  have  been 
and  are  currently  engaged  in  several  DOD-sponsored  research  projects  requiring  highly  challenging  nu¬ 
merical  simulations  of  complex  flows  that  have  never  been  attempted  before.  We  have  been  numerically 
simulating  transitional  and  turbulent  flow  phenomena  in  a  variety  of  flow  geometries,  such  as  wall  jets 
(funded  by  AFOSR),  supersonic  wakes  (supported  by  ARO),  and  boundary  layers  along  flat  and  curved 
walls  (supported  by  ONR).  In  all  cases,  the  flows  considered  are  strongly  time  dependent  and  the  phe¬ 
nomena  to  be  investigated  are  highly  nonlinear.  Flow  speeds  are  from  low  (incompressible)  to  high  (su¬ 
personic). 

In  all  the  projects,  Direct  Numerical  Simulations  (DNS)  and  Large-Eddy-Simulations  (LES)  are  carried 
out  with  the  goal  of  pushing  the  relevant  Reynolds  number  to  as  high  a  value  as  currently  feasible  with 
available  supercomputers.  The  DNS  are  based  on  solving  the  complete  Navier-Stokes  equations  with  no 
further  assumptions  other  than  those  made  for  deriving  these  equations.  The  LES  require  subgrid-scale 
models.  For  LES,  the  grid  spacing  is  such  that  only  the  larger  scales  are  resolved  directly  by  the  computa¬ 
tional  grid.  Scales  smaller  than  the  grid  have  to  be  modeled. 

In  all  the  projects,  the  underlying  computational  model  was  “spatial”.  Thus,  the  structures  evolving  in  the 
transitional  and  turbulent  flows  could  develop  (grow  or  decay)  not  only  in  time,  but  also  in  the  spatial 
(downstream)  direction,  as  is  the  case  in  realistic  laboratory  experiments  or  in  free  flight.  This  spatial 
model  is  in  contrast  to  the  so-called  “temporal”  model,  for  which  spatial  (downstream)  periodicity  is  as¬ 
sumed  and  which  therefore  introduces  assumptions  that  are  not  realistic  for  many  situations.  The  temporal 
model  allows  for  short  computational  domains  and  thus  considerably  reduces  the  computational  demands 
and,  in  particular,  the  size  of  the  data  files  produced  by  the  numerical  simulations.  The  realistic  spatial 
simulation  model,  on  the  other  hand,  leads  to  considerably  higher  computational  complexity  because  of 
the  typically  much  larger  (in  the  downstream  direction)  computational  domains  required  and  because  of 
the  difficulties  associated  with  providing  and  implementing  proper  inflow  and  outflow  boundary  condi¬ 
tions. 

As  a  consequence  of  employing  the  spatial  model,  considerably  more  time  and  effort  were  necessary  for 
code  development.  The  numerical  methods  had  to  be  very  accurate  and  efficient  for  simulations  with  such 
large  grid  sizes  to  be  at  all  feasible  on  supercomputers.  These  efficiency  and  accuracy  requirements,  to¬ 
gether  with  the  difficulties  associated  with  inflow  and  outflow  conditions,  required  extensive  testing  in 
the  development  phase  of  the  computer  codes.  For  this  reason,  adequate  pre-processing  capabilities  have 
been  essential  for  effective  utilization  of  available  supercomputing  resources.  The  high-performance  mul¬ 
tiprocessor  system  acquired  (Origin  2000)  has  met  our  current  pre-processing  needs.  The  pre-processing 
capability  of  this  compute  server  has  enabled  us  to  do  all  of  our  pre-processing  locally.  This  has  helped  to 
relieve  the  network  and  to  unclog  the  supercomputers.  After  all,  supercomputers  are  best  utilized  for 
number  crunching,  and  using  it  for  code  development  purposes,  which  can  be  done  more  efficiently  on 
considerably  less  expensive  machines,  is  in  fact  a  waste  of  supercomputing  resources. 

In  addition,  post-processing  of  the  massive  amount  of  data  that  results  from  our  simulations  had  been  very 
cumbersome  (if  possible  at  all)  when  carried  out  remotely  at  the  supercomputing  centers.  Even  attempting 
it  had  been  wasteful  of  supercomputing  resources.  Our  post-processing  needs  have  been  met  more  effi¬ 
ciently  and  cost  effectively  by  the  powerful  high-performance  Origin  2000  in  combination  with  the  newly 
acquired  AYS  flow-visualization  software  (purchased  with  the  current  DURIP  grant)  which  runs  on  our 
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graphics  workstation  (SGI  Power  Onyx  purchased  with  a  previous  DURIP  grant).  The  flows  that  we  have 
been  studying  are  dominated  by  the  generation  and  evolution  of  coherent  turbulent  structures.  These 
three-dimensional,  non-periodic  structures  cover  a  wide  range  of  spatial  and  temporal  scales.  Due  to  their 
non-periodic  nature,  their  dynamics  cannot  be  analyzed  by  means  of  Fourier  transforms.  Therefore,  de¬ 
tailed  investigations  of  the  space-time  evolution  of  the  physical  flow  field  have  been  essential  to  gaining 
an  understanding  of  the  dynamics  and  the  underlying  physics  of  the  turbulent  structures.  This  has  impor¬ 
tant  implications  for  the  post-processing  of  the  data  generated  in  our  numerical  simulations. 

Since  it  is  the  rapid  change  of  the  flow  field  with  time  that  is  important,  the  most  revealing  information 
concerning  the  dynamical  behavior  of  the  flow  structures  has  been  obtained  by  scrutinizing  animations  of 
the  unsteady  data,  rather  than  snapshots  of  the  flow  depicting  its  state  at  one  instant  in  time.  Conse¬ 
quently,  our  post-processing  demands  in  terms  of  disk  space,  CPU  time,  and  computer  memory  are  much 
greater  than  for  “mainstream”  CFD  applications,  where  only  a  steady-state  mean  flow  is  computed  and 
visualized.  Typical  data  sets  require  approximately  50MB  of  disk  space  (and  50MB  of  memory)  per  time 
step,  where  a  meaningful  animation  of  a  non-periodic  flow  requires  several  hundred  time  steps.  With  the 
powerful  Origin  2000,  in  combination  with  our  SGI  Power  Onyx,  we  are  now  able  to  generate  these  an¬ 
imations  very  quickly. 

The  typical  procedure  is  to  transfer  the  large  data  sets  (usually  several  GB)  overnight  from  the  DOD 
supercomputers  to  our  large  local  disks.  Also,  for  later  scrutinization,  the  data  is  frequently  archived  on 
our  newly  acquired  DLT  7000  tape  drive  which  holds  about  30  GB  of  data.  Depending  on  the  animation, 
portions  of  the  large  data  sets  are  then  further  processed  on  the  Origin  2000,  e.g.,  Fourier  transformed  to 
physical  space,  analyzed  using  Proper  Orthogonal  Decomposition  (which  is  extremely  memory  and  CPU¬ 
time  intensive  and  is  only  feasible  on  the  Origin  2000).  From  these  pre-processed  data,  time-dependent 
animations  are  then  generated  using,  for  example,  the  AVS  visualization  software  on  our  SGI  Power 
Onyx  (which  can  now  be  dedicated  almost  exclusively  to  graphics  processing).  For  generating  the  time- 
dependent  animation  with  AVS,  at  every  time  step,  the  data  first  is  read  from  disk  into  memory.  Then,  the 
data  is  processed  to  extract  useful  information,  e.g.,  to  compute  level  surfaces  of  flow  variables,  particle 
traces,  kinetic  energy  levels,  entropy  distributions,  etc.  Finally,  the  processed  data  is  sent  to  the  graphics 
processor  for  display  on  the  screen.  This  whole  process  is  then  repeated  for  the  next  time  step.  Due  to  the 
fast  disk  access  and  the  sophisticated  hardware/software,  this  process  takes  place  interactively,  e.g.  the 
angle  of  view  or  the  displayed  surface  level  can  be  changed  on  the  fly  while  the  animation  is  running. 
Thus,  our  current  post-processing  setup  allows  us  to  cut  the  time  required  to  generate  a  time-dependent 
animation  (including  processing  on  the  Origin  2000)  from  a  day  to  less  than  an  hour  and,  in  some  cases, 
even  to  several  minutes.  This  tremendous  speedup  has  resulted  in  a  qualitative  change  in  our  post¬ 
processing:  The  shorter  turn-around  time  has  enabled  us  to  do  truly  interactive  graphics  processing  of  sig¬ 
nificant  data  sets. 

A  great  advantage  of  the  acquired  Origin  2000  system  is  that  additional  processors  and  memory  can  be 
added  as  our  pre-  and  post-processing  demands  increase  proportionally  with  the  increased  computing 
powers  of  new  supercomputers  at  the  DOD  High-Performance  Computing  Centers.  This  enables  us  to 
solve  ever  larger  CFD  problems,  pushing  advancements  in  turbulence  modeling  and  turbulence  research. 


